EVALUATING PERFORMANCE OF AN ARTIFICIAL INTELLIGENCE TOOL TO DENTIFY INDIVIDUALS IN NEED OF THREE-YEAR COLORECTAL CANCER POST-POLYPECTOMY SURVEILLANCE

Introduction: Post-polypectomy surveillance aims to increase early detection and prevention of colorectal cancer (CRC). High risk neoplasia (HRN), defined as adenoma or sessile serrated polyp (SSP) greater than 1 centimeter, adenoma with tubulovillous or villous histology, high-grade dysplasia, >5 adenomas or SSPs, or traditional serrated adenoma, increase CRC risk and require a 3-year surveillance interval as per the 2020 United States Muti-Society Task Force (MSTF) guidelines. We previously developed and implemented a natural language processing (NLP) algorithm that uses artificial intelligence (AI) to extract data from colonoscopy and pathology reports to determine guideline-concordant surveillance intervals. In this study, we aimed to evaluate the NLP’s false negative rate for 3-year surveillance to help assess its overall performance before broad application in a health system intervention.

Methods: The setting for this study is a large academic health center with a defined primary care population, robust referral-based care, and 6 outpatient endoscopy facilities that perform over 15,000 screening colonoscopies annually. We randomly selected 1100 patients with screening colonoscopy exams performed between 1/1/2022 and 12/21/2022. We excluded patients with diagnostic colonoscopies. We compared the NLP-assigned post-polypectomy surveillance intervals to manual chart review with MSTF intervals as the gold standard. All discrepancies were reviewed by two physicians. We then calculated the false negative rate of the NLP model for the subgroup of patients requiring 3-year surveillance.

Results: Our overall study population consisted of 967 patients (50.4% female, 68.7% non-White, and mean age 57.2 ± 11.5 years). There were 104 patients (10.7%) with a HRN diagnosis by manual chart review, compared to 121 patients (12.5%) by NLP. In total, 7 HRN patients (6.7%) were not detected by the NLP (false negative rate). In these 7 missed cases, 4 were misclassified by the NLP to longer surveillance intervals, 2 cases were misclassified to 1-year follow up, and 1 case could not be classified by NLP. Compared to manual chart review, the NLP was 93.3% sensitive and 97.2% specific for identifying patients indicated for 3-year surveillance.

Conclusion: We successfully developed an NLP algorithm in a large health system that uses AI to automatically identify patients with HRN for 3-year surveillance colonoscopy. The false negative rate was 6.7%. Given that timely colonoscopy surveillance is essential for early detection and removal of polyps, particularly for this high-risk group of individuals, this study highlights the potential of AI-based tools to promote guideline-based colonoscopy surveillance intervals and improve CRC outcomes. Future goals are to use this tool to automate recall and scheduling of HRN patients due for surveillance in our health system.